Order-Adaptive Entropy Gating + BackoffNgramMixer + Drift-Free TTT

val_bpb: 0.5466 (3-seed mean, std 0.0010) | ~15.99 MB | 8×H100 SXM

Adds order-adaptive entropy gating on top of PR #779's BackoffNgramMixer + Drift-Free TTT submission. Instead of using a single entropy center for all n-gram orders, each order gets its own threshold — higher orders are trusted at lower entropy, lower orders only kick in when the model is more uncertain.

Results (8×H100 80GB SXM, PyTorch 2.9.1+cu128)

What Changed vs PR #779

PR #779 uses a single entropy_center=3.5 for all n-gram orders. We replace this with per-order entropy centers:

# PR #779 (single entropy center for all orders)
alpha = alpha_min + (alpha_max - alpha_min) * sigmoid(2.0 * (entropy - 3.5))

# This submission (per-order entropy centers)
ent_centers = {7: 3.0, 6: 3.2, 5: 3.5, 4: 3.8, 3: 4.2, 2: 4.5}
ent_center = ent_centers[matched_order]
alpha = alpha_min + (alpha_max - alpha_min) * sigmoid(2.0 * (entropy - ent_center))

Higher-order n-grams (7, 6, 5) are trusted at lower model entropy — when the model is fairly confident, the precise n-gram correction refines the prediction. Lower-order n-grams (4, 3, 2) only intervene at higher entropy — when the model is confused enough that even coarse statistics help.

This is an eval-time-only change. It modifies how existing n-gram statistics are combined with neural predictions, not when data enters the cache. The n-gram cache is still updated strictly AFTER scoring each batch (score-first).

Legality

• Score-first: N-gram cache updated AFTER scoring each batch. No future tokens leak into predictions.
• No oracle selection: Alpha depends only on model entropy and n-gram order, not on ground truth.
• Artifact size: All seeds strictly under 16,000,000 bytes (max: 15,995,959).
• Training time: Capped at 600s (10 min) on 8×H100 (actual: ~582s).
• Eval time: TTT eval ≤607s on 8×H100.

Ablation

Credits

• BackoffNgramMixer + Drift-Free TTT + Base model: PR #779
• Order-adaptive entropy gating: This submission